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(57) Abstract: An apparatus and method for process- 
ing pose data derived from a pose of an elongate object 
such as, for example, a jotting implement, a pointer, a 
robotic arm or a cane. The elongate object has a tip 
contacting a plane surface with one or more invariant 
features. The pose of the elongate object is measured 
optically from on-board by an optica] measuring sys- 
tem with the aid of the invariant feature. The pose is 
used for preparing a corresponding pose data and a sub- 
set of the pose data is identified and transmitted to an 
application such as a user application, where the sub- 
set can serve as command data or input data. Since 
the elongate object moves while its tip is contacting the 
surface the pose is measured periodically at sufficiently 
frequent measurement times ti to describe the motion at 
a desired temporal resolution. The subset can include 
all or a portion of the orientation data that describe the 
orientation of the elongate object in space and/or posi- 
tion data of the tip on the surface. The position can be a 
relative position of the lip with respect to any feature or 
its previous position, or an absolute position in world 
coordinates. The subset can also contain a mix of ori- 
entation and position data. 
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(57) ABSTRACT 

An apparatiLS and method for processing pose data derived 
from a pose of an elongale object such as, for example, a 
jolting implement, a pointer, a robotic arm or a cane. The 
elongate object has a tip contacting a plane surface v/ilh one 
or more invariant features. The pose of the elongale object 
is measured optically from on-board by an optical measuring 
system with the aid of the iovariant feature. The pose is used 
for preparing a corre.sponding pose data and a subset of the 
pose data is identified and transmitted to an application such 
as a user appUcation, where the subset can serve as com- 
mand data or input data. Since the elongate object moves 
while its lip is coniaciing the surface the pose is measured 
periodically at sufBcicnily frequent measurement limes \. to 
describe the motion at a desired temporal resolution. The 
subset can include all or a portion of the orientation data lhal 
describe the orienlalion of ihe elongate object in space 
and/or position data of Ihe lip on the surface. ITie position 
can be a relative position of the lip with respect lo any 
feature or its previous position, or an absolute position in 
world coordinates. The subset can also contain a mix of 
orientation and position data. 
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PROCESSING POSE DATA DERIVED FROM THE 
POSE OF AN ELONGATE OBJECT 

FIELD OF THE INVENTION 

[0001] The presenl invention relates generally to an appa- 
ratus and method for processing data derived from a pose of 
an elongate object having a tip in contact with a plane 
surface. 

BACKGROUND OF THE INVENTION 

[0002] When an object moves with respect to stationary 
references such as a ground plane, fixed points, lines or 
reference surfaces, knowledge of the object's inclination 
with respect to these references can be used to derive a 
variety of its parameters of molion as well as its pose. Over 
time, many useful coordinate systems and methods have 
been developed to track the po.se of objects and to param- 
eterize their equations motion. For a theoretical background 
the reader is referred to textbooks on classical mechanics 
such as Goldstein et al.. Classical Mechanics, 3"* Edition, 
Addison Wesley 2002. 

[0003] In one specific field it is important to know the pose 
of an object to derive the position of its tip while it contacts 
a plane surface. Various types of elongate objects can benefit 
from knowledge of their pose and position of their tip, and 
more precisely the absolute position (in world coordinates) 
of their tip while it is in contact with a plane surface. These 
objects include walking canes when in touch with the 
ground, pointers when in touch with a display or projection 
surface, writing devices when in touch with a writing 
surface, and styluses when in touch with an input screen. 
[0004] llie need to determine the absolute position of the 
tip or nib is deeply fell in ihe field of input devices such as 
pens and styluses. Here, the absolute position of Ihe lip has 
to be known in order to analyze the information written or 
traced by the user on the writing surface. Numerous teach- 
ings of pens and related input devices providing relative tip 
position and absolute lip position arc discussed in Ihe prior 
art. Some of these teachings rely on inertial navigation 
devices including gyroscopes and accelerometers as 
described in U.S. Pat. Nos. 6,492,981; 6,212.296; 6,181, 
329; 5,981,884; 5,902,968. Other techniques combine iner- 
tial navigation with force sensing as described in U.S. Pat. 
Nos. 6,081,261; 5,434,371. The prior art also teaches cap- 
turing and analyzing forces applied to the pen point in U.S. 
Pal. No. 5,548,092. Still other techniques rely on Iriangula- 
tion using signal receivers and auxiliary devices on or 
adjacent to the writing surface as found in U.S. Pat. Nos. 
6,177,927; 6,124,847; 6,104,387; 6,100,877; 5,977,958; 
5,484,966. It should be noted that various forms of radiation 
including short radio-frequency (RF) pulses, infra-red (IR) 
pulses, and ultrasound pulses have been taught for triangu- 
lation and related techniques. A tew examples of yet another 
set of solutions employing digitizers or tablets are discussed 
in U.S. Pal. Nos. 6,050,490; 5,750,939; 4,471,162. 
[0005] The prior art also addresses the use of optical 
systems to provide relative, and in some cases, absolute 
position of Ihe tip of a pen or stylus on a surface. For 
example, U.S. Pal. No. 6,153,836 teaches emilling two light 
beams from the stylus to two receivers lhal determine angles 
wiih respecl to a two-dimensional coordinate system defined 
within the surface. The tip position of the stylus is found 



with the aid of these angles and knowledge of the location 
of the receivers. U.S. Pat. No. 6,044,165 teaches integration 
of force sensing at the tip of the pen with an optical imaging 
system having a camera positioned in Ihe world coordinates 
and looking al Ihc pen and paper. Slill other teachings use 
optical systems observing Ihe lip of Ihe pen and its vicinity. 
These teachings include, among others, U.S. Pal. Nos. 
6.031,936; 5,960.124; 5,850,058. According to another 
approach, the disclosure in U.S. Pat. No. 5,103,486 proposes 
using an optical ballpoint in the pen. More recently, optical 
systems using a light source directing light at paper have 
been taught, e.g., as described in U.S. Pat. Nos. 6,650,320; 
6,592,039 as weU as WO 00217222 and U.S. Pat. Appl. Nos. 
2003-0106985; 2002-0048404. 

[0006] In some prior art approaches the writing surface is 
provided with special markings that Ihe optical system can 
recognize. Some early examples of pens using special mark- 
ings on the writing surface include U.S. Pat. Nos. 5,661,506; 
5,652,412. More recently, such approach has been taught in 
U.S. Pat. Appl. 2003-0107558 related literature. For still 
fiirther references, Ihe reader is referred to U.S. patent 
application Ser. Nos. 10/640,942 and 10/745,371 and the 
references ciied therein. 

[0007] Most of the prior art approaches listed above are 
limited in that they yield relative position of the tip on the 
writing surface. Tablets and digitizers obtain absolute posi- 
tion but they are bulky and inconvenient. Of the approaches 
that provide absolute position of the lip without tablets by 
using optical systems, most rely on observing the relation- 
ship of markings provided on the writing surface to the tip 
of the pen. This approach is limiting it that it requires a 
specially-marked writing surface, which acts as a quasi- 
tablct. 

[0008] In addition to being cumbersome, state-of-the-art 
pens and styluses employing optical systems usually gener- 
ate a limited data set. In fact, most only provide data 
corresponding to Ihe trace Iravereed on the writing surface. 
Meanwhile, there are many applications that could benefit 
from a rich stream of data from the pen or stylus. In fact, the 
prior art indicates many situations in which interactions 
between a user employing a pen or stylus and a machine, 
e.g.. a computer, are limited. For a few examples of appli- 
cations and systems that could benefit from a richer stream 
of data from the pen or stylus Ihe reader is referred to U.S. 
Pat. No.s. 6,565,611; 6,.502,114; 6,49.3,736; 6,474,888; 
6,454,482; 6,415,X56; 6,396,481 and U.S. Pat. Appl. Nos. 
2003-0195820; 2003-0163525; 2003-0107558; 2003- 
0038790; 2003-0029919; 2003-0025713; 2003-0006975; 
2002-0148655; 2002-0145587 and U.S. Pat. No. 6.661,920. 

OBJECTS AND ADVANTAGES 

[0009] In view of the shortcomings of Ihe prior art, it is the 
object of the invention to provide an apparatus and method 
for processing pose data derived from a pose of an elongate 
object such as a jotting implement, cane, pointer or a robotic 
arm. Specifically, it is an object of the invention to provide 
for processing or pose data derived in this manner and 
idenlificnion of a subset of the pose data for use as conlrol 
data or inpul data in applications. 

[0010] 'ITiese and numerous other advantages will become 
apparent upon reading the detailed description in conjunc- 
tion with the drawing figures. 
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SUMMARY OF I'HE INVENTION 

[0011] The present invention provides a method for pro- 
cessing pose data derived from a pose of an elongate ol^ect 
whose lip is contacting a plane surface with one or more 
invariant features. In accordance with the method, the tip is 
placed on the surface and the physical pose of the elongate 
object is measured optically from on-board the elongate 
object with the aid of the invariant feature. 'ITie pose data 
corresponding to the pose is prepared and a subset of the 
pose data is identified. The subset is transmitted to an 
application such as a user application, where ihc subset can 
serve as command data or input data. The elongate object 
can undergo a motion while iLs lip is contacting a surface. 
Thus, in a preferred embodiment the method calls tor 
periodically measuring Ihe pose data al measurement limes 
I, such that the pose data al successive measurement limes i, 
can be used to describe the motion al a desired temporal 
re.solution. 

[0012] The subset can include ail or a portion of orienia- 
tion data that describe the orientation of the elongate object 
in space. The orientation data can include an inclination 
angle e or any angle or set of angles describing orientation 
in suitable coordinates, e.g., polar coordinates. Alternatively, 
the sub.sel can include all or a portion of position data of the 
lip on the surface. The position can be a relative position of 
the tip relative to any feature including one or more of the 
invariant features or an absolute position of the tip on the 
surface in world coordinates. The subset can also contain a 
mix of orientation and position data. 
[0013] In one embodiment, the orientation of the elongate 
object is described by Euler angles and the subset of the pose 
data includes at least one Euler angle. In fact, in this 
embodiment the inclination angle 9 can be simply Ihe 
.second Euler angle. In addition, the subset can contain a roll 
angle ip (third Euler angle) and a rotation angle ^ (first Euler 
angle), llie orientation data contained can be used as any 
type of input. For example, the orientation data can represent 
control data that is used for executing commands in the 
application or input data that is entered into the application 
or simply stored in an appropriate format. Of coivse, the 
position data can also be used as any type of input, including 
control data and input data. It should be noted, thai Ihe 
subset can also contain all of the pose data, e.g., when the 
application is a motion-capture application. 
[0014] The invariant features can be permanent or tempo- 
rary, and spatial or temporal. The plane surface can be a 
jotting surface, such as a paper surface, a screen, a tablet, a 
pad or any other type of surface on which a user can perform 
a jotting operation. In this embodiment Ihe elongate object 
is preferably a jotting implement such as a pen, pencil or 
stylus. In general, the elongate object can also be a pointer, 
a robotic tirm or a cane. In fact, the elongate object Ls any 
object that whose pose can be used to derive input data. 
[0015] The invention further provides an apparatus for 
processing pose data describing the motion of the elongate 
object whose tip is contacting Ihe surface. The apparatus has 
a measuring arrangement for optically measuring the pose 
from on-board the elongate object. The measurement can be 
periodic at measurement limes t; and the periodicity of this 
measurement is chosen depending on the application and 
resolution, e.g., temporal resolution, of pose and pose data 
required. The apparatus has a processor for preparing the 



pose data corresponding to the pose and identifying a subset 
of the pose data. A communication link is provided for 
transmitting the subset to an application. 
[0016] llie processor is preferably mounted on the elon- 
gate object. In applications where considerable data pro- 
cessing is performed by the processor, the processor can be 
remote. The communication link is preferably a wireless 
communication link. 

[0017] The subset separated from the pose data by the 
processor can be sent to the application for further process- 
ing the subset, e.g., using it as input data, by using the same 
or a different communication link, depending on the location 
of the host rtinning the user application for which the suKsel 
is used as input data. For example, the bust is a computer and 
the application is a data file. In this case the subset can 
contain input data into the data file. Alternatively, the host is 
a digital device and Ihe user application is an executable file 
for executing a command and ihe subset contains control 
data. The application can also be a molion-captuic apphca- 
tion for capturing the motion of the elongate object, or a 
trace-capture application for capturing the trace de.scribed by 
the lip of the elongate object on the surface during the 
motion. 

[0018] The details of the invention will now be described 
in detail with reference to the drawing figures. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0019] FIG. 1 is a three-dimensional view of an apparatus 
of the invention illustrating the motion of an elongate object 
whose lip is contacting a surface. 

[0020] FIG. 2A-C are diagrams illustrating the Euler 
rotation convention as used herein. 

[0021] FIG. 3 is a block diagram illastrating the format- 
ting or preparation of the pose data into subsets. 

[0022] FIG. 4 is a three-dimensional view of an elongate 
object with an imaging system for measuring Ihe pose. 
[0023] FIG. S is a three-dimensional view of an elongate 
objecl with a scanning system for measuring Ihe pose. 
[0024] FIG. 6 is a three-dimensional view of a preferred 
application in which the elongate object is a jotting imple- 
ment for jotting on a plane surface. 

[0025] FIG. 7 is a diagram illustrating the jotting imple- 
ment communicating pose data to remote devices via com- 
munication links. 

[0026] FIG. 8 is a block diagram illustrating the uses of 
command and input data derived from the pose data of the 
jotting implement of FIG. 7. 

[0027] FIG. 9 is a diagram illustrating still another 
embodiment of the apparatus of invention. 

DETAILED DESCRIPTION 
[0028] The present invention will be best understood by 
first referring to FIG. 1 illustrating an exemplary apparatus 
10 for processing pose data 12 derived from a pose of an 
elongate objecl 14. Elongate object 14 moves while its tip 16 
is in contact with a plane surface 18. Apparatus 10 processes 
pose data 12 thai describe the pose of elongate object 14 at 
a number of measurement times. Thus, pose data 12 
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describes ihe motion thai elongale objecl 14 executes or is 
made to execute by a user while tip 16 is in contact with 
surface 18. A sensor 20, e.g.. a piezoelectric element or any 
other suitable sensor can be used to ascertain when tip 16 is 
contacting surface 18. 

[0029] Elongate object 14 is any type of generally elon- 
gated object whose pose when objecl 14 is stationary or in 
motion yields useful pose data 12. For example, elongate 
object 14 is a cane used for walking support and surface 18 
is a walking surface, or elongate objecl 14 is a robotic arm 
and surface 18 is a work surface. In still other cases, elongate 
object 14 is a jotting implement, such as a pen, pencil or 
stylus and surface 18 is a jotting surface. In the present 
embodiment elongale object 14 is a pointer and surface 18 
is a screen or a tracing surface. 

[0030] Elongate objecl 14 has an on-board measuring 
arrangement 22 for optically measuring its pose with the aid 
of one or more invariant features 32, 34, 36 on surface 18. 
In the present case all features 32, 34, 36 arc all on surface 
18. In general, however, it is also pos.sible to use feamres 
that are not in the plane of surface 18 if there is a .sufhcienl 
number of features on surface 18. 

[0031] In the present embodiment, invariant features are 
an edge 32, a reference point 34 and a surface structure 36. 
lliese invariant features are merely exemplary of the types 
of features that can be temporarily or permanently associ- 
ated with surface 18 and be used for measuring the pose of 
object 14. Invariant features 32, 34, 36 are used in deriving 
a relative or absolute position of tip 16 on surface 18 and for 
measuring the remaining portion of the pose, i.e., ihe ori- 
entation of pointer 14. Preferably, the positions of invariant 
features 32, 34 and 36 arc defined in world coordinates 
(X„,Y<,,ZJ. Furthermore, if possible, the location of invari- 
ant features 32, 34 and 36 is preferably such that al least a 
subset of them is visible to arrangement 22 for all poses that 
object 14 is expected to assume. 

[0032] A number of optical measurement methods using 
on-board arrangement 22 to recover pose of object 14 can be 
employed. In any of these methods arrangement 22 uses 
on-board elements to obtain pose data 12 in accordance with 
any well-known pose recovery technique including geomet- 
ric invariance, triangulation, ranging, path integration and 

[0033] In Ihe preferred embodiment arrangement 22 i.s an 
optical measurement arrangement such as an imaging sys- 
tem or a scanning system mounted on pointer 14 for on- 
board determination of the pose with reference to one or 
more of invariant features 32, 34, 36 on surface 18. 

[0034] Apparatus 10 has a processor 26 for preparing pose 
data 12 corresponding to the pose of pointer 14 and for 
identifying a subset 48 of pose data 12 required by an 
application 28. Specifically, application 28 uses subset 48 
which may contain all or less than all of pose data 12. Note 
that processor 26 can be located on pointer 14 or be remote, 
as is the case in this embodiment. 

[0035] A communication link 24 is provided for sending 
pose data 12 to application 28. Preferably, communication 
link 24 is a wireless communication link established with the 
aid of a wireless transmitter 30 mounted on pointer 14. In 
embodiments where processor 26 and application 28 are 
mounted on pointer 14, communicatioD link 24 can be an 



electrical connection. In still other embodiments, commu- 
nication hnk 24 can be a wired remote link. 

[0036] During operation a user 38 holds pointer 14 in 
hand. User 38 places tip 16 of pointer 14 on .surface 18 with 
invariant features 32, 34, 36 and executes a movement such 
thai pointer 14 executes a motion 40. For better visualiza- 
tion, motion 40 is indicated in dashed lines 42, 44 that mark 
the positions assumed by lip 16 and end 46 of pointer 14 
during motion 40. 

[0037] For the purposes of Ihis invention, fine 42 is 
referred to as the trace of tip 16. Also, for the purposes of the 
present invention, motion 40 is defined to end at Ihe time 
when tip 16 stops contacting surface 18. 

[0038] Motion 40 may produce no movement of end 46 or 
tip 16, i.e., no trace 42. In fact, motion 40 is not limited by 
any parameter other than that tip 16 must remain in contact 
with surface 18. Thus, changes in orientation of pointer 14 
are also considered to be motion 40, just as changes in 
position (i.e., change in x and y coordinates) of tip 16 on 
surface 18. In the present case, orientation of pointer 14 is 
described by inclination angle G, rotation angle ip and roll 
angle ij) referenced with respect to a center axis C.A. of 
pointer 14. A change in at least one of these angles consli- 
luies motion 40. 

[0039] In the present case, lip 16 touches down on surface 
18 at point 48. At the time of touch down center axis C.A. 
of pointer 14 is inclined to a surface normal Z' al inclination 
angle 6 equal to 6„. Furthermore, rotation and roll angles ij>, 
arc equal to ()>„, respectively. For convenience, in Ihe 
present embodiment angles 0, (j), are Euler angles. Of 
course, other angles can be used to describe the orientation 
of pointer 14. In fact, a person skilled in the art will 
appreciate that any convention for describing the rotations of 
pointer 16 can be adapted for Ihis description. For example, 
Ihe four Caylyle-KIein angles or quaternions can be 
employed. 

[0040] FIGS. 2A-C illustrate a convention for describing 
the orientation of pointer 14 using the Euler angles. Pointer 
14 has a length I measured from tip 16 at the origin of 
non-roiaied object coordinates pC'.Y'.Z') as .shown in FIG. 

2A. 

[0041] Center axis C.A. is collinear with Ihe Z' axis and it 
passes through lip 16 and the origin of non-rotated object 
coordinates (X',Y',Z'). In the passive rotation convention 
used herein objects coordinates will be attached to pointer 
14 while pointer 14 is rotated from initial upright position. 

[0042] Now, FIG. 2A illustrates a first counterclockwise 
rotation by first Huler angle <|i of object coordinates (X',Y',Z') 
about the Z' axis. This rotation of the objecl coordinates does 
not affect Ihe Z' axis so once rotated Z" axis is collinear with 
non-rotated Z' axis (Z"=Z'). On the other hand, axes X' and 
Y' are rotated by first Euler angle ^ to yield once rotated axes 
X" and Y". 

[0043] FIG. 2B illustrates a second counterclockwise 
rotation by second Euler angle 0 applied to once rotated 
object coordinates (X",Y",Z"). This second rotation is per- 
formed about the once rotated X" axis and therefore it does 
not aSFeci the X" axis (X"'=.X"). On the other hand, axes Y" 
and Z" are rotated by second Euler angle 0 to yield twice 
rotated axes Y'" and Z'". This second rotation is performed 
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in a plane 11 containing once rolaled axes Y", Z" and twice 
rotated axes Y ", Z ". Note that axis C.A. of pointer 14 is 
rotated counterclockwise by second Euler angle 0 in plane 
n and remains collinear with twice rolaled axis Z'". 

[0044] A third counterclockwise rotation by third Euler 
angle i|> is applied to twice rolaled object coordinates (X'", 
Y'", Z'") as shown in FIG. IC. Rotation by is performed 
about twice rolaled axis Z'" that is already collinear with 
object axis Z rotated by all three Euler angles. Meanwhile, 
twice rotated axes X'", V" are rotated by \|) to yield object 
axes X.Y rotated by all three Euler angles. Object axes 
X,Y,Z rotated by all three Euler angles .)), 8 and ij) define 
Euler rotated object coordinates (X,Y,Z). Note that tip 16 of 
pointer 14 remains at the origin of all object coordinates 
during the Euler rotations. 

[0045] Now, referring back to FIG. 1, the pose of pointer 
14 includes its orienlalioo, i.e., Euler angles (41, 6, ip), and 
position of lip 16, i.e., the coordinates (x,y,z) of the point ai 
which lip 16 contacts surface 18. For convenience, ihc 
orientation of pointer 14 and position of lip 16 are expres.sed 
in world coordinates (X^.Y^.Z^,). World coordinates (X„,Y„, 
Z;,) have a world origin (0,0,0) that can be used to describe 
an absolute position of tip 16 on surface 18. In fact, world 
coordinates (X^.Y^.Z;,) can be used for an absolute measure 
of any parameler(s) of the po.se of pointer 14. Alternatively, 
any paramctcr(s) of the pose of pointer 14 can be described 
in ii relative manner, e.g., with reference to non-stationary or 
relative coordinates (X^.Y^.Z,) or .simply with respect to the 
previous pose. 

[0046] To describe the absolute pose of pointer 14 it is 
convenient 10 relate Euler rotated object coordinates describ- 
ing the orientation of pointer 14 to world coordinates 
(X„,Yo,Z„). To do this, one notes that the orientation of 
object axis Z' in world coordinates (X„,Y<„ZJ prior to the 
three Euler rotations is normal to plane (X^.Y^). Second 
Euler angle 6 defines the only counterclockwise rotation of 
object coordinates that is not about an object Z axis (this 
.second rotation is about the X"°X"' axis rather than axis Z', 
Z" or Z'"). Thus, Euler angle 6 is an inclination angle 6 
between the completely Euler rotated object axis Z or axis 
C.A. and original object axis Z', which is normal to plane 
(Xo,Y„) al the point of contact of lip 16. 

[0047] Optical mea.suring arrangement 22 measures the 
pose of pointer 14 during motion 40 al measurement times 
ti and processor 26 prepares corresponding pose data 12. 
Pose data 12 consists of measured values of parameters 
(4>,B,i|>,x,y,z) al measurenaent limes tj. Invariant features 32, 
34, 36 whose positions are defined in world coordinates 
(Xo,Yo,Zo) are employed by optical measuring arrangement 
22 to express pose data 12 in world coordinates (X,„Y,„Z„) 
. The frequency of ihe periodic mea.surements depends on 
the use of pose data 12 and desired performance, e.g., 
temporal resolution. It should be noted that periodic mea- 
surement is not limited to any predetermined lime or fre- 
quency schedule. In other words, the times between any two 
successive measurements of the pose can be arbitrary. Pref- 
erably, however, arrangement 22 measures the pose al a 
frequency that is high enough to obtain pose data 12 that 
describe motion 40 at the temporal resolution required by 
application 28. 

[0048] Wireless transmitter 30 of communication link 24 
sends pose data 12 or parameters ((^,6,Tii,x,y,z) collected at 



measurement limes t, to processor 26. Pose data 12 can be 
transmitted continuously, in bursts, in parts, al arbitrary or 
preset limes or as otherwise desired. Processor 26 prepares 
a subset 48 of pose data 12, for example the absolute 
position (x,y) of lip 16 and sends it to application 28. 
Application 28 uses absolute position (x,y) of tip 16 at 
mea.surement times t; to chart trace 42 of tip 16 on surface 
18 as pointer 14 executes motion 40. In other words, unit 28 
recovers trace 42 corresponding to Ihe movement of tip 16. 
Note that Ihe resolution of trace 42 recovered by unit 28 can 
be improved by increasing the number of pose measure- 
mems or increasing the frequency of measuremcni limes t,. 
It should be noted that pose data 12 should be formatted for 
appropriate communications between transmitter 30, pro- 
cessor 26 and application 28. Any suitable communication 
and formatting standards, e.g., IEEE interface standards, can 
be adapted for these purposes. For specific examples of 
formatting standards the reader is referred to Rick Poyner, 
LGC/Telegraphics, "Wintab™ Interface Specification: 
16-bit and 32-bil API Reference", revision of May 9, 1996; 
Universal Serial Bus (USB), "Device Class Definition for 
Human Interface Devices (HID)", Firmware Specification, 
USB Implementers' Forum, Jun. 27, 2001 and six-degree of 
freedom interface by Ulrica Larsson and Johanna Pettersson, 
"Developmcnl and evaluation of a 6D0F interface to be 
used in a medical application". Thesis. Linkopings Univer- 
sity, Department of Science and Technology, Sweden, Jun. 
5, 2002. 

[0049] The remaining pose data 12, i.e., (()),6,>J),z) can also 
be used in Ihc pre.senl embodiment. Specifically, processor 
26 can prepare addilional subsets or send all of the remain- 
ing pose data as a single subset to application 28 or to a 
different application or device serving a different function. 
Any mix of orienialion and position data derived from pose 
data 12 can be used in subset 48. In fact, in some embodi- 
ments processor 26 keeps all pose data 12 in subset 48 such 
lhal all pose data 12 is used by application 28. 'Iliis is done 
when application 28 has to reconstruct the entire motion 40 
and not just trace 42 of lip 16 on surface 18. For example, 
this is done when application 28 includes a motion-capture 
application. Once again, the temporal resolution of motion 
40 can be improved by increasing the frequency of mea- 
surement limes t(. Note lhal in this ca.se parameters of pose 
data 12 lhal vary slowly are ovcrsampled. 
[0050] It should also be noted that surface 18 is plane and 
hence the value of parameter z does not change. Thus, z can 
be set al a constant value, e.g., z=0, and left out of pose data 
12 to reduce the amount of data that needs to be transmitted 
by transmitter 30. 

[(N)S1] In FKJ. 3 a block diagram illasirates the process- 
ing of pose data 12 by processor 26 and its use by application 
28 in more detail. In a first step SO. pose data 12 is received 
by processor 26 via communication link 24. In a second step 
52, processor 26 determines which portion or subset 48 of 
pose data 12 is required. This selection can be made based 
on application 28. For example, when application 28 is a 
trace-capture application that charts trace 42, then only 
position data of tip 16, i.e., (x.y) need to be contained in 
subset 48. On the other hand, when application 28 is a 
motion-capture application, then all pose data 12 need to be 
contained in subset 48. 

[0052] In step 58 all pose data 12 is selected and passed to 
a subset formaiiing or preparing step 60A. In step 60A pose 
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dala 12 is prepared in ihc lorn) of subsel 48A as required by 
application 28. For example, pose dala 12 is arranged in a 
particular order and provided with appropriate fooler, header 
and redundancy bits (not shown), or as otherwise indicated 
by dala porting standards such as those of Rick Poyncr, 
LGC^elegraphics (op. cit.). 

[0053] In step 62, only a portion of pose data 12 is 
selected. Three exemplary cases of partial selection arc 
shown. In the first case, only position data is required by 
application 28. 

[0054] Hence, in a step 59B only position data (x,y,z) is 
selected and the remaining pose data 12 is discarded. In a 
subsequent step 606, position data (x,y,z) is prepared in the 
form of subset 48B as required by application 28 and/or as 
dictated by the porting standards. In the second case, in a 
step 59C, only orientation dala ((t>,8,ap) is selected and the 
rest of pose data 12 is discarded. Then, in a step 60C, 
orientation data (i(),9,i|)) is prepared in the form of a subset 
48C for use by application 28. In the third case, in a step 
S9D, a mix of pose data 12. including some position data and 
some orientation data are selected and processed corre- 
spondingly in a step 60D to prepare a subset 48D. 

[0055] A person skilled in the art will appreciate that the 
functions described can be shared between processor 26 and 
application 28, e.g., as required by the system architecture 
and data porting standards. For example, some preparation 
of subsel 48 can be performed by application 28 upon 
receipt. It should also be noted that in some embodiments 
pose data 12 can be pre-processed by transmitter 30 or 
post-processed al any point before or after preparation of the 
corresponding subset 48 in accordance with any suitable 
algorithm. For example, a statistical algorithm, such as a 
least squares fit can be applied to pose data 12 derived al 
different measurement limes Ij or to successive subsets 48. 
Furthermore, quantities such as time derivatives of any or aU 
parameters of pose data 12, i.e.. 



dx dy dz d<p d9 dip^ 



[0056] can be computed. Also, various sampling tech- 
niques, e.g., oversampling can be used. 
[0057] Subset 48 is transmitted to application 28 via a 
communication channel 72. Application 28 receives subsel 
48 as an input that is treated or routed according to its use. 
For example, in a step 64, subset 48 is used as control data. 
Thus, subset 48 is interpreted as an executable command 66 
or as a part of an executable command. On the other hand, 
in a step 68, subset 48 is used as input data and saved to a 
data file 70. 

[0058] In one embodiment, application 28 passes infor- 
mation to proces.sor 26 to change the selection criteria for 
subsel 48. Such information can be passed via communica- 
tion channel 72 or over an alternative link, e.g., a feedback 
link 74. For example, apphcation 28 requests subsel 48A to 
be transmitted and uses subsel 48Aas input data for dala file 
70. At other times, application 28 requests subset 48C to be 
transmitted and uses subsel 48C as command data for 
executable command 66. Alternatively, processor 26 can 
indicate a priori whether any subset 48 should be treated as 



input dala or control dala. In still another alternative, user 38 
can indicate wilh the aid of a separate apparatus, e.g., a 
switch mounted on pointer 14 (not shown), whether subset 
48 is intended as control dala or input data. A person skilled 
in the art will recognize that there exist a large number of 
active and passive methods for determining the interpreta- 
tion and handling of data being transmitted in subset 48 by 
both processor 26 and application 28. 
[0059] In general, ihe optical measuring performed by 
on-board optical measuring arrangement can be imple- 
mcolcd in a number of ways. For example, FIG. 4 illustralcs 
an elongate object 100 wilh a tip 102 contacting a plane 
surface 104 endowed wilh an invariant feature 106. Invariant 
feamre 106 is a polygon of known orientation, size and 
position on surface 104. Note that only a portion of object 
100 is indicaled for belter visualization. Object 100 is 
aligned along the Z axis, which is collinear wilh a center axis 
C.A. that is used for reference lo world coordinales (not 
shown). 

[0060] An opiical measuring sysiem 108 is mounied on 
object 100 for performing on-board opiical mea,suremenLs of 
pose. In fact, system 108 is an imaging sysiem or image 
capturing sysiem for oplically measuring Ihe pose of object 
IfliO using invariani fealure 106. Imaging sysiem 108 has a 
lens 110 and an image capluring device 112 in the form of 
an electronic optical sensor or pixel array positioned in an 
image plane 114 defined bi^ lens 110. Preferably, lens 110 
has a wide field of view' ft and a .substantially .single 
viewpoint for preserving 3-D perspective information of 
nearby surroundings. Lens 110 can include various optics 
including refractive, reflective and/or caladioptric optics 
accompanied by opiical relays, mirrors, apertures, field 
flallcners, image guides and other elements, as will be 
appreciated by one skilled in the an. In fact, lens 110 should 
be .selected as appropriate for imaging .surface 104. 
[0061] Array 112 has imaging pixels 116 positioned in 
image plane 114 described by orthogonal axes X', Y'. These 
axes are parallel lo axes X and Y of rotated object coordi- 
nates. The Eulcr angles from non-rotated to rotated object 
coordinates are as indicated. 

[0062] During operation, radiation 118 such as sunlight or 
artificial illumination is incident on surface 104. A scattered 
portion 118' of radiation 118 travels lo elongate object 110 
at an angle of incidence 8, to central axis C.A. More 
precisely, scattered portion 118' such as that propagating 
along path 120 scattered from a point Pp of a comer of 
invariani feature 106 arrives within solid angle 6 and is 
imaged by imaging sysiem 108 on image plane 114 at image 
point P'. Scattered portion 118' from point Pp and from the 
remainder of fealure 106 and plane 104 or their portions 
carries wilh il image information. This .spatial intensity 
variation or image information is used to determine the pose 
of object 100 in accordance wilh any technique for recov- 
ering position from an image of surface 104 and feature 106 
produced on imaging pixels 116 of array 112. A person 
skilled in Ihe art will recognize that perspective imaging is 
parliailarly well-.suited for this purpose. For more informa- 
lion on appropriate imaging optics and methods Ihe reader is 
referred lo U.S. patent application Ser. No. 10/640,942. 

[0063] Another ahernative for implementing an on-board 
opiical measuring arrangement is shown in FIG. 5. Here, an 
elongate object 130 wilh a lip 132 contacting a plane surface 
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applications such as biotnelric applicalions requiring precise 
knowledge of Ihe motion pen 200, e.g., lo derive a biometric 
of hand 228, more frequent measuremenl times tj, e.g., in 
excess of 100 llz can be used. In particular, such precise 
knowledge can be required when the biometric application 
is a user verification application. In another embodiment, 
application 224 i.s a trace-capture application for capturing 
traces 230A, 230B and 230C marked by pen 200 on paper 
206. More precisely, application 224 is an on-line or off-line 
handwriting recognition application requiring that measure- 
ment times tj be on the order of 100 Hz (At°Vioo s) to 
properly recognize handwriting. Another application 224 
may require that a distance As that nib 202 moves on paper 
surface 204 between successive measurements be on Ihe 
order of Vioo of an inch or about 0.12 mm. In still another 
embodiment, the trace-capture application can be a signature 
verification applicatioo that can require higher precision 
than handwriting recognition. It should be noted that Ihe 
biometric application can also perform user identification 
and/or signature verification. In the present embodiment, 
application 224 is an on-line handwriting recognition appli- 
cation and measurement times tj are repeated at a frequency 
on the order of 100 Hz. 

[0074] During operation the user moves band 228 trans- 
lating to a motion of pen 200 resulting in ink traces 230 on 
surface 204 of paper 206. Traces 230 are produced when nib 
202 contacts surface 204 at a writing point during Ihe 
motion. Optical measuring arrangement 214 measures the 
pose of pen 200 periodically at measurement times t,. with 
the aid of invariant features 216 while traces 230 are being 
produced. 

[0075] A first writing point P„j„, sometimes referred to as 
pen-down, and a last writing point P_, sometimes referred 
to as pen-up, are indicated on trace 230A which nib 202 
produced during a motion 232 of pen 200. Also indicated is 
an intermediate writing point P„„, al a location al which nib 
202 was located during a !q>ecific measuremenl time l„. 'l\vo 
writing points at successive measurement times t, and t^.^, 
are also shown and, for illustrative purposes, a time duration 
At between them. Trace 230B is shown in the process of 
being written by the user. Trace 230C has already been 
written, and two writing points at succe.s.sive mea.surement 
times t„ and t^.) are shown separated by a distance As. 
[0076] Application 224 is an on-line handwriting recog- 
nition application and so the frequency of measurement 
times tj between pen-down and pen-up points for each trace 
230A. 230B and 230C is on the order of 100 Hz such that 
time duration At is about Vioo s. Therefore, pose of pen 200 
is measured by arrangement 214 with a temporal re-solution 
of about Vioo 

[0077] Processor 218 prepares pose data corresponding lo 
the succession of poses assumed by pen 200 in motions 
executed between pen-down and pen-up points. Processor 
218 also identifies the pose data this is required in subset 220 
to run application 224. In some handwriting recognition 
applications, subset 220 need only contain the absolute 
positions of nib 202 al measurement times t; corresponding 
to writing points P^j. In other words, subset 220 only 
contains position data of nib 202 on surface 204 rather than 
orientation data of pen 200. Furthermore, the locations of 
writing points P^j are expressed by corresponding vectors Di 
in global coordinates (Xo,Yo,Z„) having an origin al the lop 
right comer of paper 206. 



[0078] In an alternative embodiment, handwriting recog- 
nition application 224 requires orientation data in subset 
220, For example, an on-hne handwriting recognition appli- 
cation 224 requires inclination angle 6 lo be contained in 
subset 220. Note that inclination angle S is also the second 
Euler angle. Still other handwriting recognition application 
224 requires .subset 220 lo contain more or even all orien- 
tation data, i.e., all three Euler angles. In fact, processor 218 
can be informed by application 224 of its particular require- 
menU at any given time with respect lo subset 220, e.g., via 
communication link 226. In cases where subset 220 is to 
contain additional information, such as first or higher-order 
derivatives of any combination of or all position and orien- 
tation data, e.g., .second order derivatives 



^ ^ 



[0079] these can either be computed by processor 218 and 
included in subset 220, or they can be computed by appli- 
cation 224, as convenient. 

[0080] In an alternative embodiment, pen 200 is a .stylus 
that leaves no traces 230. In other words, nib 202 is a point 
that makes no markings. In this embodiment surface 204 
does not have to be a paper surface; it can be replaced by any 
plane surface on which jolting can be performed. Otherwise 
the method is the same as described above. 

[0081] KIC 7 illustrates another embodiment of an appa- 
ratus 248 for processing pose data derived from a pose of an 
elongate object 250 having a lip 252. Object 250 is a stylus 
whose tip 252 does not produce traces. Object 250 has a 
housing 254 containing an optical measuring arrangement 
and a processor (not shown). A wireless transmitter 262 is 
provided for communication. 

[0082] Tip 252 is placed on a plane surface 256 with 
invariant features 258A, 258B, 258C, and 258D. The mea- 
suremenl arrangement relies on features 258 for periodically 
measuring the pose of stylus 250 at measurement times t, 
when tip 252 contacts surface 256. The processor prepares 
pose dala corresponding lo the pose and idenlilies a subset 
260 to be transmitted. In the present case, subset 260 
contains all pose dala and is transmitted after each measure- 
menl lime Ij with a time stamp, also referenced by tj. Position 
data is expressed in Ihe form of a vector in world 
coordinates (X„,Y„,ZJ selected by the optical measuremenl 
arrangeraenl. Orienlation data is expressed in Euler angles. 
[0083] Transmitter 262 transmits subset 260 via commu- 
nication links 264A, 264B, 264C and 264D to various 
devices having resident applicalions requiring subset 260. 
Specifically, link 264A connects lo a network 266 that in 
turn connects lo a computer 268, e.g., a personal computer 
that nins an application 270 requiring subset 260. Network 
266 can be any type of network, including the internet, a 
local area network (LAN), a telephone network or any 
network capable of transmitting subset 260. Link 264B 
connects to a local host 272, e.g., a host computer, which is 
in communication with computer 268 via web 274. It should 
be noted that application 270 can be shared between com- 
puter 268 and local host 272, or each can use set 260 tor its 
own separate apphcation. Altcmalivcly, local host 272 can 
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serve as a relay computer only. Link 264C connects directly 
to computer 268 and may be a shorl-dislance link, e.g., a link 
which is only active when stylus 250 is operating in prox- 
imily to computer 268. Link 264D connects to a device 276 
running another application 278. For example, device 276 is 
a personal digital assistant (PDA) or a cellular telephone. In 
faci, link 264D can be an infrared link or an ultrxsound link, 
in which case a corresponding iransrailler is used to replace 
wireless transmitter 262. It should be noted that stylus 250 
can communicate via any combination of links 264 with any 
device it needs to be in communication with at any given 
time. The devices may use links 264 to communicate their 
availability and subset preferences to stylus 250 at any time. 

[0084] During operation, stylus 250 can include in subset 
260 absolute position data or vector expressed in world 
coordinates. Alternatively, stylus 250 can include in subset 
260 relative position data or vector in relative coordi- 
nates PC„Y„Z,). In still another alternative, subset 260 
includes relative position data with respect to the previous 
position, i.e., vector ADxD.i-D^.j. That is because some 
application; require only knowledge of the relative position 
of tip 252. It should also be noted that the processing of pose 
data and identification of subset 260 is a task that can be 
shared between pen 250 and any other devices. In fact, the 
processor required for processing the pose to derive the pose 
data and identify subset 260 can reside entirely on another 
device, e.g., on computer 268. 

[INIK5] Vm. 7 is a block diagram illustrating a few exem- 
plary uses of command and input data from subset 260 uf 
jotting implement 250. In a first step 280, subset 260 is 
received by either a local host or a network via communi- 
cation link 264. If subset 260 is intended for a remote host, 
then it is forwarded to the remote host in a step 282. 

[0086] In a second .step 284, a processor in the intended 
host (local host or remote host, as the case may be) deter- 
mines the requirements for subset 260. 'fhis selection can be 
made based on an intended application 300. For example, 
when application 300 only requires the parameters already 
contained in subset 260, then subset 260 is forwarded to step 
286 for preparation and direct use. Alternatively, when 
application 300 requires additional parameters, subset 260 is 
forwarded to step 288 for derivation of these additional 
parameters. 

[0087] For example, the additional parameters are deriva- 
tives of one or more of the parameters in subset 260. Thus, 
subset 260 is sent to a differentiation module 290 and then 
to a preparation module 292 for supplementing subset 260 
with the derivatives. In the example shown, lime derivatives 
of Euler angles (f and 6 are required and thus, supplemented 
and prepared subset 260' contains these time derivatives. 
[0088] Alternatively, statistical information about one or 
more of the parameters in subset 260 are required. Thus, 
subset 260 is sent to a statistics module 294 and then to a 
preparation module 296 for supplementing subset 260 with 
the statistical information. In the present example, the sta- 
tistical information is a standard deviation of second Euler 
angle 8. Thus, supplemented and prepared subset 260" 
contains the 

[0089] A person skilled in the art will appreciate that the 
functions described can be shared between local and remote 
hosts as well as application 300, e.g., as required by the 



system architecture and data porting standards. For example, 
some preparation and supplementing of subset 260 can be 
performed by application 300 upon receipt. 
[0090] Subset 260 is transmitted to application 300 for use 
as an input that is treated or routed according to its use. For 
example, in a step 302, subset 260' is used as control data. 
Thus, subset 260' is interpreted as an executable command 
304 or as a part of an executable command and used in an 
executable file 310. On the other hand, in a step 306, subset 
260" is u.sed as input data and saved to a data file 308. 
[0091] In one specific application, application 300 is a 
trace-capture application and subset 260' is used as control 
data in accordance with .standard Boolean logic. Specifically, 
apphcation includes a Boolean logic symbol reformatting 
function to translate the control data into a Boolean logic 
command, e.g., OR. AND or XOR. 
[0092] FIG. 9 is a diagram illustrating still another 
embodiment of the apparatus of invention. In this embodi- 
ment an elongate object 320 is a jotting implement 
employed on a jotting surface 322 with invariant features 
324A and 324B. Jotting surface 322 can be a screen, a pad, 
a paper surface or any other convenient surface from which 
jotting implement 320 can determine pose with the aid of an 
on-board optical measuring arrangement in accordance with 
the above leaching. Jotting implement 320 is a pen, pencil or 
a stylus. 

[0093] A u.ser 326 employs jotting implement 320 by 
placing its tip 328 on surface 322 and executing motions to 
generate pose data. Pose data is processed into subsets and 
transmitted to a variety of user devices that include, but are 
not limited to, a mainframe computer 330, a joystick 332, a 
cellular telephone 334, a personal computer 336 and a 
personal digital assistant 338. Each of these devices uses the 
appropriate portion of the .subset. For example, joystick 332 
strips all parameters other than Euler angles ^, ti and uses 
these as control data. Alternatively, joystick 332 retains 
Euler angle \J) and uses it as control data to emulate an 
activation button function. On the other hand, cellular tele- 
phone 334 uses Euler angles to select dial numbers and 
position data as control data to execute a dial command. 

[0094] It should be noted that the elongate object can be 
any type of elongate device whose physical pose can yield 
useful data. Thus, although the above examples indicate that 
the elongate object Is a jotting implement, pointer, cane, or 
robotic arm other elongate objects can be used. Also, the 
subset identified form the pose data can be supplemented 
with various additional data that may be derived from other 
devices that are or are not on-board the elongate object. 
[0095] Furthermore, the pose data and/or data in the subset 
can be encrypted for user protection or other reasons, as 

[0096] It will be evident to a person skilled in the art that 
the present invention admits of various other embodiments. 
Therefore, its scope should be judged by the claims and their 
legal equivalents. 

1 . A method for processing pose data derived from a pose 
of an elongate object having a tip, said method comprising: 

a) placing said tip on a plane surface with an invariant 
feature; 
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b) measuring said pose optically from on-board said 
elongate objeci using said invariant feature; 

c) preparing pase data corresponding to said pose; 

d) identifying a subset of said pose data; and 

e) transmitting said subset to an application. 

2. The method of claim 1, wherein said elongate object is 
undergoing a motion and said measuring is performed 
periodically at measurement limes t,, such that said pose data 
describes said motion. 

3. ITie method of claim 1, wherein said subset comprises 
orientation data. 

4. The method of claim 3, wherein said subset comprises 
an inclination angle 6. 

5. The method of claim 3, wherein said subset comprises 
at least one Euler angle. 

6. The method of claim 3, wherein said orientation data is 
used as input selected from the group consisting uf control 
data and input data. 

7. The method of claim 2, wherein said subset comprises 
position data of said tip on said surface. 

8. The method of claim 7, wherein said position data 
comprises a position of said tip relative to said invariant 
feature. 

9. The method of claim 7, wherein said position data 
comprises an absolute position of said tip on said surface. 

10. The method of claim 7, wherein said position data 
comprises a position of said tip relative to a previous 
absolute position of said tip on said surface. 

11. The method of claim 7, wherein said position data is 
used as an input selected from the group consisting of 
control data and input data. 

12. The method of claim 1, wherein said plane surface 
comprises a jotting surface and said elongate object is a 
jotting implement. 

13. The method of claim 12, wherein said jotting surface 
is selected from the group consisting of a screen, a tablet, a 
pad and a paper surface. 

14. The method of claim 1, wherein said elongate object 
is selected from the group consisting of jotting implements, 
pointers, robotic amis and canes. 

15. The method of claim 14, wherein said jotting imple- 
ments are selected firom the group consisting of pens, pencils 
and styluses. 

16. An apparatus for processing pose data derived Croni a 
pose of an elongate object having a tip, said apparatus 

comprising: 



a) a plane surface for placing said tip thereon, said plane 
surface having an invariant feature; 

b) an optical measuring means for optically mea.suring 
said pose from on-buard said elongate object using said 
invariant feature; 

c) a proce."5.sor for preparing pose data corresponding to 
said pose and identifying a subset of said pose data; and 

d) a communication link for transmitting said subset to an 
application. 

17. The apparatus of claim 16, wherein said optical 
measuring means comprises a periodic measurements means 
for periodically measuring said pose at measurement times 

18. The apparatus of claim 16, wherein said processor is 
mounted on said elongate objeci. 

19. The apparatus of claim 16, wherein said communica- 
tion link i.s a wireless communication link. 

20. The apparatus of claim 16, wherein said elongate 
objeci is selected from the group consisting of jotting 
implements, pointers, robotic arms and canes. 

21. The apparatus of claim 20, wherein said jotting 
implements arc selected from the group consisting of pens, 
pencils and styluses. 

22. The apparatus of claim 16, wherein said application 
comprises a data lile and said subset comprises input data to 
said data file. 

23. The apparatus of claim 16, wherein said application 
comprises an executable file and said subset comprises 
control data to said executable file. 

24. The apparatus of claim 16, wherein said application 
comprises a motion-capture application. 

25. Ihe apparatus of claim 24, wherein said motion- 
capture application comprises a biometric application. 

26. The apparatus of claim 25. wherein said biometric 
application comprises a user verification application. 

27. The apparatus of claim 16, wherein said application 
comprises a trace-capture application. 

28. Tlie apparatus of claim 27, wherein said trace-capture 
application comprises a handwriting recognition applica- 

29. The apparatus of claim 27, wherein said trace-capture 
application comprises a signature verification application. 

30. The apparatus of claim 27, wherein said trace-capture 
application comprises a Boolean logic symbol reformatting 
fiu notion. 



